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The Effect of Changing Grading Criteria on Student Writing 

Darlene E. Crone-Todd, Heidi L. Eyre, Scott A. Hutchens, J. Reid Jones, and Joseph J. Pear 

Students enrolled in a senior-level psychology course used a computer-aided personalized system 
of instruction (CAPSI) to write tests, and give and receive feedback on their writing at a mastery level. 
Weekly measures of the number of unit tests requested, the number of unit tests cancelled, and informal 
student feedback all indicate that a change in the feedback to allow revision of answers increased student 
persistence and activity in CAPSI. 
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In Spring 2004, Delta State University surveyed freshmen and seniors using the National Survey 
of Student Engagement (NSSE, 2000; see Bridges & Kuh, 2004). The results of the NSSE report 
indicated many significant strengths of Delta State, including high levels of student-student and student- 
faculty interaction and many positive aspects of writing (e.g., preparing several drafts, and receiving 
prompt feedback). However, the NSSE results did indicate that students at Delta State engage in less 
writing than students at comparable institutions. As suggested by Tierney and Shanahan (1991), there is a 
strong link between writing and thinking. Thus, one area that Delta State has targeted is to increase is the 
amount of writing that students produce during their four-year undergraduate degrees. 

The current study began as a way to increase student reading and writing skills within an upper- 
level psychology course at Delta State University. This course incoiporates stress on the written word 
through use of a computer-aided personalized system of instruction (CAPSI, see Pear & Crone-Todd, 
1999). With its basis in Keller’s (1968) personalized system of instruction (PSI), CAPSI incorporates 
many of the features of the original system (Grant & Spencer, 2003), including: (a) the written word as 
the main means of learning the material; (b) mastery criteria for passing unit tests prior to continuing on 
in the testing; (c) self-paced method for students to progress at their own speed; (d) use of proctors, tutors, 
or peer reviewers to provide feedback and help administer the program; (e) lectures as motivational aids, 
rather than as the main means of transmitting information. 

The growth of PSI-taught courses coincided with research indicating its clear effectiveness as a 
pedagogical tool that increased performance well over the previous gains using lecture or discussion- 
based learning (Buskist, Cush, & DeGrandpre, 1991; Johnson & Ruskin, 1977; Kulik, Kulik, & Cohen, 
1979). This work showed that PSI students learned more than students taught using conventional 
methods. When compared with other methods, PSI courses were perceived by students as more 
demanding, yet were also rated as more favorable in quality and enjoyment. 

Despite this early advantage for PSI, the system fell out of favor for a variety of reasons. One of 
the key reasons was due to the amount of time for instructors to set up and administer these courses 
(Buskist et al, 1991). However, recent advances in technology have resurrected an interest in the PSI 
methodology. In particular, CAPSI (Pear & Crone-Todd, 1999; Pear & Kinsner, 1988) has been used for 
the last few decades at the University of Manitoba to teach several undergraduate courses in psychology, 
and is now being made available through the internet to serve locations throughout Canada and the United 
States. 


CAPSI incorporates the principles of PSI by focusing on small units of study. The students are 
provided with open-ended study questions that correspond to each unit. Students are allowed to proceed at 
their own pace to master the units on tests based on these study questions. These unit tests are reviewed 
by the instructor or two peer-reviewers to determine if mastery is met. Feedback on each question is 
provided, and if the answers to all questions meet mastery criteria, the test is evaluated as a pass. If any of 
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the questions do not meet mastery criteria, then the test is evaluated as a restudy. However, there is no 
penalty for a restudy: The same unit is simply retested as described above until mastery criteria are met 
before the student may move on to the next unit. During the last few years, research on CAPSI has 
centered on the development of critical, or higher-order, thinking skills (HOT; see Crone-Todd & Pear, 
2001; Crone-Todd, Pear, & Read, 2000; Pear, Crone-Todd, Wirth, & Simister, 2001); the quality of, and 
compliance with, feedback on unit tests (Martin, Pear, & Martin, 2002a;b); and the theoretical 
underpinnings of CAPSI in terms of constructivistic and behavioristic approach to pedagogical 
philosophy and application (Pear & Crone-Todd, 2002). 

A key part of the learning in C APSI takes place through the use of feedback between students and 
more-able peers. But what is feedback? Peterson (1982) suggests that feedback can take on many 
different usages that are not well-defined. In CAPSI, feedback on the outcome of a unit test has often 
been in the form of a restudy or pass. In any case, we likely expect that through the processes involved in 
feedback that our students should develop some sort of rules (Martin & Pear, 2003) or heuristics, with 
which they can solve problems or answer questions in the future. Through the judicious use of shaping 
and fading procedures from the verbal community for following these rules appropriately, students learn 
to emit mastery-level performance. The processes involved in shaping and fading in the process of 
learning is what social constructivists term “scaffolding” in “constructing knowledge” (Pear & Crone- 
Todd, 2002). The restudy was meant to provide feedback that should indicate to the student what needed 
to be revised to attain mastery on that unit. While the restudy has typically been seen by both educators 
and researchers as a non-punitive outcome for answers that did not meet criteria simply because the 
student could try again (e.g., Keller, 1968), if the feedback were to result in students requesting and 
submitting less unit tests, then from a behavioral perspective, the restudies would necessarily be defined 
as punitive in nature. 

During the term in which the first author taught a CAPSI-assisted senior-level psychology course, 
low levels of student persistence indicated that the restudies may be serving a punitive function. Thus, 
changes in the feedback provided to students for unit test mastery were incorporated. The purpose of this 
research, then, is to determine the pedagogical effectiveness of these changes. 

Method 


Participants 

The participants were 14 undergraduate psychology students (12 females and 2 males; 29% 
African-American, and 71% Caucasian) enrolled in a senior-level Applied Behavior Analysis course at 
Delta State University. All fourteen of these student gave signed informed consent, and these students are 
included in the study. 

Materials and Equipment 

A dedicated website was designed by the first author to serve as a place for students to download 
all course materials, including a general course manual which provided syllabus content and detailed 
information about how to use CAPSI, and the unit study questions. 

CAPSI (http://www.webcapsi.com) ran on a University of Manitoba server, and was used to 
administer and record all aspects of the PSI-related course activities, including unit tests, peer reviewing, 
and email. Computers were available on the Delta State campus in labs and in the library for students’ 
access, or they could login from any location that has an internet connection. 

Design and Procedure 
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The design used in this research is an interrupted time series design (Goodwin, 2005), in which 
measures of student behavior are taken both before and after an event that occurred in the middle of the 
series (i.e., change in mastery criteria in the middle of the tenn). The independent variable is the 
conditional result, which was either not available, or available. The dependent variables include the 
number of students who requested tests, number of cancelled tests, number of tests passed, number of 
restudies, and the percentage of students who attempted unit tests, per week in the course. 

As part of the course procedures, students were instructed in how to use CAPSI to take unit tests 
within a 60 minute time period. Upon request, CAPSI would randomly select three questions from a 
given unit and present them to the student. The student then typed and saved their answer in a text box 
under each question. When complete, the student could cancel (if they did not feel adequately prepared) 
or submit the test for review. 

A submitted test is sent for anonymous review to either two peer reviewers (students who had 
previously mastered the given unit) or to the instructor. If the test was sent to two peer reviewers, both 
had to independently agree that the test was a “pass”. Otherwise, it was a restudy. The instructor could 
render a “pass” or “restudy” individually. “Mastery criteria” was defined as having complete and correct 
answers to all questions. Each unit must be passed before the student may move on to the next unit. 
Students may appeal any restudy decision through CAPSI by providing a good argument for why their 
answer was correct. The appeal would then be sent to the instructor, who can render a decision. 

There were 1 5 unit tests, with the first unit test based on the course manual and the rest based on 
the course textbook (Martin & Pear, 2003). Students were encouraged to work through the unit tests on a 
weekly basis. By the eleventh week of class, a decline in unit test writing was observed, so the 
“conditional pass” was incorporated another alternative. An instruction was given to all students that if a 
test was near mastery, then a “conditional” pass would be given, and that the student could use the appeal 
function to provide a revised answer to the instructor. 

After final grades were calculated and submitted at the end of the term, the data were analyzed 
using the following method. The total number of unit tests attempted during the four full course weeks 
preceding, and following, email and an in-class announcement indicating that a conditional pass would be 
available. In addition, the number of unit tests cancelled for each student, by week, were recorded. As a 
measure of consumer satisfaction, qualitative results are reported based on classroom observations and 
communication with students in the course. 


Results 


Quantitative Results 

Figure 1 displays the total number of requested tests by week in course. Note that for 1 1 of the 
students, that there was a statistically significant increase ( t = 3.03 , p = < .005, 1-tailed) in the number of 
tests requested after the conditional was introduced (M = 6.00, SD = 4.51) when compared to the four 
weeks prior to the announcement (M = 2.57, SD = 2.3 1). Figure 2 shows the number of cancelled tests 
reduced in number after the conditional was introduced (M = 0.93, SD = 1.27) when compared with the 
four weeks prior to the announcement (M = 1.86, SD = 2.18). Note that 1 1 of the students either reduced, 
or kept constant, the number of cancelled tests. It is also important to realize that this was not just a select 
group of students increasing their writing activity: a greater proportion of students attempted tests after 
the conditional pass was added. Taken together, the decrease in cancelled tests and the increase in 
requested unit tests provides evidence that the restudy results may have been serving as punishers, rather 
than as feedback to encourage changes in test answers. In addition, the conditional decision appears to 
have served as a less punitive way to encourage revised writing. 
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Figure 1. The total number of requested tests, by student, during the four full weeks before and 
after the conditional pass was introduced. 
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Figure 2. The total number of cancelled tests, by student, during the term before and after the 
conditional pass was introduced. 
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Qualitative Results 

Statements from students up to the announcement of the conditional pass result indicated that 
they were concerned about being able to write at a mastery level, or that the restudies were perceived as 
being very stressful. After the introduction of the conditional pass, students started to report liking the 
program better, and that they wished the conditional decision had been available all along. These 
observations confirmed the students’ test-taking behavior in the course was increasing due to the change 
in feedback. 


Discussion 

The quantitative and qualitative results suggest that the use of a conditional decision may be an 
effective tool for helping students provide mastery-level answers in higher education courses. The 
addition of the conditional pass coincided with a dramatic increase in the self-paced test-taking activity, 
reduced number of cancelled tests, and positive self reports on the part of the students. Therefore, it 
appears that the basis for this study is confirmed: the restudy was serving as a punitive consequence for 
below-mastery textual responses. 

While it may seem unconventional to use a conditional pass in a mastery-based system, previous 
iterations of CAPSI used such a procedure. What is clear from the data is this: The increase in student 
test-taking is an indication of increased persistence (response rate) and student engagement (increased 
response rate and less avoidance), which is linked to increased performance in PSI courses. By possibly 
providing students a less punitive environment in which to write and revise, while still maintaining 
mastery criteria, CAPSI was utilized in a way that for many students has the potential to increase 
persistence in developing their writing and thinking skills. 

The use of feedback provides information that the answer is “close” is similar to the process 
involved in “shaping” (Martin & Pear, 2003), in which successive approximations to a target behavior are 
reinforced. In the case of the conditional, we might think of the consequence involving some praise as a 
reinforcer for being closer to mastery, and that the subsequent revision in the writing is one more 
approximation to a mastery-level set of responses. Using these behavioral procedures, we can adapt them 
to environments such as CAPSI or other media to help improve student performance via feedback from 
more-able peers and instructors regarding what is required to provide mastery-level answers. 

It should be pointed out that not all students are initially successful in this system. A few students 
still had some difficulty achieving mastery. The reasons for this are not entirely clear, but some 
observations should be noted here. First, some students tend to procrastinate in starting to write their unit 
tests. Second, some students appeared to have difficulty learning from or incoiporating feedback into 
subsequent answers. For example, it was observed that feedback provided on a particular answer 
involving a restudy result was ignored on subsequent unit test, where the same question appeared. In fact, 
the student’s answer on this test was exactly the same as on the previous test. This indicates that perhaps 
the student did not agree with the feedback, did not know how to incorporate the feedback, or simply did 
not read it. If it was the case that they did not agree with the feedback, then an option was to appeal the 
result. By appealing the result, they may have obtained more information to help them in the course. If 
they did not know how to incorporate the feedback, or did not read the feedback, then more research on 
how to encourage students to ask for help needs to carried out. 

Clearly, using CAPSI and other PSI-like programs that utilize the power of the internet provides 
an exciting and fertile ground in which to conduct future educational research. With its stress on the 
written word, CAPSI and other PSI-based methods can be of great assistance in campus-wide projects 
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whose aim is to increase the amount and quality of student writing. As Ski nn er (1959) pointed out, 
writing is a form of textual (verbal) behavior that is shaped through a process involving the writer and 
reviewers. As behavior analysts increasingly work with colleagues, administrators, and students who are 
not familiar with the behavioral paradigm, it is important to learn to conceptualize (speak) in a way that 
make communication with others possible. As such, the three paradigms (cognitive, constructivist, and 
behavioral) synthesize into combined efforts to increase the pedagogical effectiveness of these tools, the 
future looks bright for using the underlying behavioral processes effectively to increase student 
motivation to attempt and learn how to write at the higher level of education. 


References 

Bridges, B. K., & Kuh, G. D. (2004). The National Survey of Student Engagement: A new indicator for 
postsecondary accountability. In R. Kazis, J. Vargas, & N. Hoffman (Eds.) 

Double the numbers: Increasing postsecondaiy credentials for underrepresented youth. 
Cambridge, MA: Harvard Education Press. 

Buskist, W., Cush, D., & DeGrandpre, R. J. (1991). The life and times of PSI. Journal of Behavioral 
Education, 1, 215 - 234. 

Crone-Todd, D. E., & Pear, J. J. (2001). Application of Bloom’s Taxonomy to PSI. Behavior Analyst 
Today, 3,204-210. 

Crone-Todd, D. E., Pear, J. J., & Read, C. N. (2000). Operational definitions for higher order thinking 
objectives at the post-secondary level. Academic Exchange Quarterly, 4(3), 99-106. 

Facione, P.A. (1997). Critical thinking: What it is and why it counts. Located on the California Academic 
Press Web site at URL: http://www.calpress.com/critical.html. 

Grant, L. K., & Spencer, R. E. (2003). The personalized system of instruction: review and applications to 
distance education. Retrieved September 25, 2005 from the International Review of Research in 
Open and Distance Learning website: http://www.irrodl.Org/content/v4.2/grant-spencer.html 

Goodwin, C. J. (2005). Research in psychology >: methods and design (4rd ed.). Hoboken, NJ: John Wiley 
& Sons. 

Halpem, D. F. (1998). Teaching critical thinking for transfer across domains. American Psychologist, 53, 
449-455. 

Johnson, K. R., & Ruskin, R. S. (1977). Behavioral instruction: An evaluative review. Washington, DC: 
American Psychological Association. 

Keller, F. S. (1968). "Good-bye teacher ..."Journal of Applied Behavior Analysis, 1, 79-89. 

Kulik, J., Kulik, C. L. C., & Cohen, P. A. (1979). A meta-analysis of outcome studies of Keller’s 
Personalized System of Instruction. American Psychologist, 34, 307 - 318. 

Martin, G., & Pear, J. (2003) Behavior modification: What it is and how to do it (7h ed.). Upper Saddle 
River, NJ: Prentice Hall. 

Martin, T. L., Pear, J. J., & Martin, G. L. (2002a). Feedback and its effectiveness in a computer-aided 
personalized system of instruction course. Journal of Applied Behavior Analysis, 35, 427-430. 


40 


The Behavior Analyst Today 


Volume 8, Issue 1, 2007 


Martin, T. L., Pear, J. J., & Martin, G. L. (2002b). Analysis of proctor marking accuracy in a computer- 
aided personalized system of instruction course. Journal of Applied Behavior Analysis, 35, 309- 
312. 

National Survey of Student Engagement. (2000). NSSE 2000 Overview, Indiana University Center for 
Postsecondary Research and Planning, Bloomington, IN. 

Pear, J. J., & Crone-Todd, D. E. (1999). Personalized system of instruction in cyberspace. Journal of 
Applied Behavior Analysis, 32, 205-209. 

Pear, J. J., & Crone-Todd, D. E. (2002). A social constructivist approach to computer-mediated 
instruction. Computers & Education, 38, 221-231. 

Pear, J. J., Crone-Todd, D. E., Wirth, K. M., & Simister, H. D. (2001) Assessment of thinking levels in 
students' answers. Academic Exchange Quarterly, 5(4), 94-98. 

Pear, J. J., & Kinsner, W. (1988). Computer-aided personalized system of instruction: An effective and 
economical method for short- and long- distance education. Machine-Mediated Learning, 2, 213- 
237. 

Peterson, N. (1982). Feedback is not a new principle of behavior. Behavior Analyst, 5, 101-102. 

Skinner, B. F. 1957. Verbal behavior. New York: Appleton-Century-Crofts. 

Tierney, R. J., & Shanahan, T. (1991). Research on the reading-writing relationship: interactions, 
transactions, and outcomes. In R. Barr, M. L. Camil, P. Mosenthal, & P. D. Pearson (Eds.), 
Handbook of Reading Research (vol. 2, pp. 246-280). New York: Longman. 

Author contact information: 

Darlene E. Crone-Todd, Ph.D. 

Div. of Counselor Education & Psychology, 

DSU Box 3142, 

Delta State University, 

Cleveland, MS 38732. 

Email: dctodd@deltastate.edu 


Heidi L. Eyre 
Department of Psychology 
Jacksonville State University 
700 Pelham Road, 

Jacksonville, AL, 36265-1602 
Email: heyre@jsu.edu 

Scott A. Hutchens 

Division of Counselor Education and Psychology 

Delta State University 

Ewing Hall 335 

DSU Box 3142 

Cleveland, MS 38733 


41 


The Behavior Analyst Today 


Volume 8, Issue 1, 2007 


Email: shutchen@deltastate.edu 
J. Reid Jones 

Division of Counselor Education and Psychology 

Delta State University 

Ewing Hall 335 

DSU Box 3142 

Cleveland, MS 38733 

Email: rjones@deltastate.edu 

Joseph J. Pear 
Department of Psychology 
University of Manitoba 
190 Dysart Rd 
Winnipeg, MB R3T 2N2 
Canada 

Email: pear@cc.umanitoba.ca 


Advertising in the Behavior Analyst Today 


Advertising is available in The Behavior Analyst Today. All advertising must be 
paid for in advance. Make your check payable to Joseph Cautilli. The ad copy should be in 
our hands at least 3 weeks prior to publication. Copy should be in MS Word or Word 
Perfect, RTF format and advertiser should include graphics or logos with ad copy. 

The prices for advertising in one issue are as follows: 

1/4 Page: $50.00 1/2 Page: $100.00 Full Page: $200.00 

If you wish to run the same ad in both issues for the year, you are eligible for the 
following discount: 

1/4 Pg.: $40 - per issue 1/2 Pg.: $75 - per issue Full Page: $150.00 - per issue 
An additional one-time layout/composition fee of $25.00 is applicable 


For more information, or place an ad, contact Halina Dziewolska by phone at (215) 
462-6737 or e-mail at: halinadz@hotmail.com 


42 


